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Using the Mass Storage System at ZIB within I3HP 

H. Stiiben’^* and S. Wollny'^ 

®'Konrad-Zuse-Zentrum fiir Informationstechnik Berlin, Takustr. 7, 14195 Berlin, Germany 

In the framework of I3HP there are two Transnational Access Activities related to Computational Hadron 
Physics. One of these activities is access to the mass storage system at Konrad-Zuse-Zentrum fiir Information¬ 
stechnik Berlin (ZIB). European lattice physics collaborations can apply for mass storage capacity in order to store 
and share their configurations or other data (see http://www.zib.de/i3hp/). In this paper formal and technical 
aspects of usage as well as the conformance to the International Lattice DataGrid (ILDG) are explained. 


1. 13 HADRON PHYSICS 

The HadronPhysics Integrated Infrastructure 
Initiative (I3HP) is a project that originates from 
a joint initiative of over 2000 experimental, the¬ 
oretical and computational physicists working in 
the held of hadron physics pQ . I3HP is funded by 
the European Commission in the Sixth Frame¬ 
work Programme. The project is structured into 
nine Transnational Access Activities, seven Net¬ 
working Activities, and twelve Joint Research Ac¬ 
tivities. There are three activities that are related 
to lattice QCD: the Networking Activity Compu¬ 
tational Hadron Physics, Transnational Access to 
supercomputer resources at NIC (Jiilich) |21, and 
Transnational Access to mass storage capacity at 
ZIB (Berlin) |3]. NIC is one of three national 
German supercomputer centres, ZIB runs the su¬ 
percomputer centre of the federal state of Berlin. 

2. TRANSNATIONAL ACCESS 

The idea of Transnational Access is to give for¬ 
eign researchers access to important, typically ex¬ 
perimental facilities. It gives experimentalists the 
opportunity to carry out interesting experiments 
at facilities that they usually cannot use. The ac¬ 
cess activities related to computational QCD have 
a similar intention and provide access to compu¬ 
tational facilities. While experimentalists have to 
travel to the corresponding laboratories, compu¬ 
tational facilities can be accessed via the internet. 


*Talk given at the Workshop on Computational Hadron 
Physics, Nicosia, Cyprus, 14-17 September 2005. 


Researchers who want to use a facility have to 
write a scientific proposal that is being peer re¬ 
viewed. How to apply for using the mass storage 
system at ZIB is explained on our web page |21. 
It is not necessary to write an application for just 
downloading configurations. For downloading, a 
certificate is required (see section EJ and special 
software has to be used (see sectional. 

3. LATTICE DATAGRIDS 

In large scale lattice QCD projects one typi¬ 
cally stores gauge field configurations and propa¬ 
gators. Gauge fields are stored permanently while 
propagators are stored for a limited period of 
time. Propagators require much more storage 
space than configurations. Hence they are kept 
at the site hosting the computer on which they 
were calculated. On the other hand, QCD gauge 
held conhgurations are smaller and, in the case 
of dynamical fermions, much more expensive to 
generate. This has lead to the idea of sharing 
conhgurations in order to fully exploit them. 

The International Lattice DataGrid (ILDG) El 
was started to make QCD gauge held conhgura¬ 
tions available at an international level. So far, 
ILDG infrastructures are being built up in Japan, 
UK, USA, and Germany. The German ILDG 
Grid is called LatFor DataGrid (LDG) [5]. ILDG 
develops standards for data formats and a com¬ 
mon middleware. The dehnitions of a metadata 
format and a binary hie format are completed (see 
El). The LatFor DataGrid conforms to the ILDG 
standards. 
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For this Transnational Access Activity we 
have decided to integrate the storage system at 
ZIB into the broader LDG and ILDG activities. 
Hence, the LatFor DataGrid became a joint effort 
of DESY (Hamburg and Zeuthen), NIC (Jiilich) 
and ZIB (Berlin). 

4. COMPONENTS OF LDG 

The main hardware components are storage el¬ 
ements (SE) that have hierarchical mass storages 
systems (HSM) attached. Each site (in Berlin, 
Hamburg, Jiilich, and Zeuthen) operates such a 
storage element. The hierarchical mass storages 
systems are large tape libraries that work with 
tape robots. The total capacity of the HSM sys¬ 
tem at ZIB is about 1.2 PetaByte. Data stored 
on the system is very safe because there always 
exist two copies on different tapes. 

The storage elements are small servers that run 
the dCache software 1^1 which was developed by 
DESY and Fermilab for storing huge amounts 
of data distributed among heterogeneous server 
nodes. From a user’s perspective the distributed 
storage element servers provide a single virtual 
filesystem tree. Data may reside in the server’s 
disk cache or might be migrated to tape. The 
dCache software performs data exchanges to and 
from the attached tape libraries automatically 
and invisibly to the user. 

Besides the storage back-ends there are user in¬ 
terfaces at the front-end. In order to set up a Grid 
infrastructure, there are several software compo¬ 
nents (middleware) needed in addition. These 
components are a virtual organisation, Grid in¬ 
formation services, a file catalogue, an access con¬ 
trol service, and a metadata catalogue. At the 
middleware level LCG-2 software is used supple¬ 
mented by developments of DESY. LCG is the 
Large Hadron Collider (LHC) Computing Grid 

m 

A virtual organisation (VO) is an organisa¬ 
tional unit in a Grid infrastructure. The VO 
representing the LatFor DataGrid is called ildg. 
Grid information services handle e.g. authentica¬ 
tion. The file catalogue maps logical filenames to 
physical locations and manages replicas of files. 
The access control service (AGS) handles access 


permissions. In LDG not all data is necessarily 
public. The ACS allows to store public and pri¬ 
vate data in the same environment. The meta¬ 
data catalogue holds the metadata and makes it 
possible to list metadata and perform search op¬ 
erations. 

5. IMPORTANT CONCEPTS 

Three important concepts for the usage of the 
hard- and software infrastructure shall be ex¬ 
plained: (1) authentication via certificates, (2) 
logical filenames and physical locations of files, 
and (3) data formats. 

5.1. Authentication 

In a Grid context users are authenticated by 
presenting a certificate, which is similar to the pri¬ 
vate/public key concept of the secure shell. The 
exact technical procedure for obtaining a certifi¬ 
cate depends on the certificate authority (CA)). 
GAs for this Grid are listed in (Hj. 

In any case, there are three basic steps that 
have to be done (cf. the documentation of your 
GA). First, one has to create a certificate request 
file and a corresponding personal key. The per¬ 
sonal key should be carefully protected with a 
secure passphrase and backed up. Second, the re¬ 
quest file has to be sent to the responsible CA in 
a secure way. In general personal authentication 
by presenting an identity card or passport is re¬ 
quired. Third, one has to install the certificate 
that one receives from the CA and the personal 
key on the machine where an LDG user interface 
(UI) is installed. 

With a valid certificate, one can use a so called 
grid-proxy, which will do all necessary authentica¬ 
tion in the background for a given period of time. 
So one does not have to type the passphrase every 
time. Usually, a certificate is valid for one year. 
In general, renewal is a much easier process than 
obtaining the initial one. 

5.2. Naming 

Table^shows examples of three types of names 
that are used in the context of LDG. This section 
explains these meaning of the names and conven¬ 
tions for forming them. 

When retrieving data (downloading gauge field 
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Table 1 

Examples of names. 


object 

name 

ensemble 
configuration 
physical location 

WWW. Iqcd.org/ildg/qcdsf/nf2_clover/b5p29kpl3500-16x32 
/grid/ildg/qcdsf/nf2_clover/b5p29kpl3500-16x32/qcdsf.515.00320.lime 
srm://dcache.zib.de/pnf s/zib.de/data/ildg/\ 

qcdsf/nf2 clover/b5p29kpl3500-16x32/qcdsf.515.00320.lime 


Table 2 

Array declarations corresponding to the storage sequence of SU(3) gauge fields. Lx, Ly, Lz, Lt are the 
extensions of the lattice, dim = 4 is the dimension of space-time, and Ncolour = 3 is the dimension of 
the SU(3) matrices, 
language array declaration 

C double U[Lt][Lz][Ly][Lx][dim][Ncolour][Ncolour][2]; 

Fortran complex U(Ncolour, Ncolour, dim, Lx, Ly, Lz, Lt) 


configurations) from the Grid one has to spec¬ 
ify a logical filename (LFN). The Grid middle¬ 
ware translates the LFN into a physical location. 
There may exist multiple copies of the data in 
the Grid, so called replicas. The middleware is 
supposed to find the best available copy. 

On uploading a configuration one has to specify 
an ID for the ensemble to which the configuration 
belongs, which is called MarkovChainllRI, and 
a physical location. The URI (unified resource 
identifier) is unique in the world. The physical 
location can be considered as an absolute path to 
the configuration file within LDG. The physical 
location is the place where the data is actually 
stored. 

Looking at the naming conventions adopted in 
LDG, one can see from the examples shown in 
Table m that all three types of names have a 
large part in common. All names contain the 
virtual organisation ildg. In the case of the 
MarkovGhainURI ildg is strictly speaking the 
last part of the URL www.lqcd.org/ildg. The 
uniqueness of the URL leads to the uniqueness of 
the MarkovGhainURI. 

After ildg follows the name of the collabora¬ 
tion which has generated the data, in the exam¬ 
ples qcdsf. The parts of the names that follow 
are chosen by the collaboration. However, the 
structure should be such that the next part of 
the names represents a project and the part af¬ 
ter that represents an ensemble. The name of a 


configuration and the physical location have a file 
name in addition. 

In the examples nf2_clover stands for the 
Nf = 2 clover improvement project. The name 
of the ensemble b5p29kpl3500-16x32 repeats the 
essential part of the metadata, i.e. (3 = 5.29, 
K = 0.13500 on a 16^ x 32 lattice. In the file name 
qcdsf . 515.00320. lime 515 is a job chain ID and 
00320 is a trajectory counter. As mentioned be¬ 
fore, these three parts of the names were freely 
chosen by the collaboration. 

Syntactically the names are composed out of 
parts separated by slashes. Basically, the phys¬ 
ical location is a Unix file name specification, 
i.e. a real directory structure is implied. In the 
framework of ILDG it would be allowed to chose 
names for ensemble, configuration, and physical 
location completely independently. For example, 
one could think of using much shorter names for 
the LFN in order to facilitate typing. However, 
in LDG it was decided to essentially use the same 
names at all levels for reasons of clarity. 

5.3. Data formats 

Data formats were defined by ILDG. There are 
conventions for formats of metadata and bi¬ 
nary data cni On uploading binary data correct 
metadata have to be supplied. 

5.3.1. Metadata 

Within ILDG, a special metadata format, 
called QGDml, for the description of configura- 
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Table 3 

Overview of Itool commands._ 

command description 

Iget Getting a binary of a configuration or the metadata for an ensemble or a configu¬ 

ration from the Grid. 


Iput 


11s 

Unit 

lupdate 

Ivalidate 


Putting a binary and the corresponding metadata on the Grid. It is not allowed to 
place a binary without corresponding (syntactically-) correct metadata. Operation 
will take place in one transaction, i.e., the data is either stored by successfully 
finishing the operation or nothing will be stored if the operation fails 

Lists all configurations of an ensemble sorted by their LFN (can also be used to 
show all ensembles in the MDG by using the —all option). 

Initialises a new ensemble in the MDG (requires administration rights). 

Updates metadata in the MDG (has to be valid QGDml) 

Gheck conformance of metadata to QGDml 


tions and ensembles has been defined QGDml 
consists of two XML schemata, one for the de¬ 
scription of an ensemble and one for the descrip¬ 
tion of a single configuration. All metadata is 
stored and exchanged in the XML format. This 
allows a formal validation before metadata is be¬ 
ing uploaded to the metadata catalogue. For ex¬ 
ample, a valid ensemble description has to have 
a <markovChainURl>-tag and must contain phys¬ 
ical and algorithmic information (see |5]). 

A valid configuration description must have a 
<dataLFN>-tag, which is the link to the binary 
file that can be downloaded from the Grid. In 
addition, a <markovChainURl>-tag has to be pro¬ 
vided, which is the link back to the ensemble that 
it belongs to. Documentation on how to markup 
configurations can be found in |^. 

5.3.2. Binary data 

A binary file format for storing SU(3) gauge 
field configurations was defined by ILDG. This 
format is described in An ILDG binary file 
consist of several parts that are packaged using 
the LIME file format, which was developed by 
SciDAG. A LIME API, utilities and documenta¬ 
tion are available from m With the API one can 
read and write LIME files from a G programme. 
Employing the utilities one can pack or unpack 
LIME files at command line level. One can also 
extract individual files. 

An SU(3) gauge field configuration packaged 


according to the conventions of ILDG contains at 
least three files. A file inside a package is called a 
record. These three files or record types, respec¬ 
tively, are called: 

ildg-format 

ildg-binary-data 

ildg-data-LFN 

The record ildg-binary-data contains the 
gauge field configuration. That record contains 
exactly the bytes of an array of floating point 
numbers as it is declared in Table El The floating 
point format is IEEE and the byte ordering is big 
endian. 

The record ildg-f ormat is an XML-document 
that contains the precision of the floating point 
numbers (32 or 64 bit) and the lattice size (see 
[TO] for the exact format). Precision and lattice 
size are also contained in the metadata. 

The record ildg-data-LFN contains the logical 
file name as it appears in the metadata. 

A convenient way to extract records (files) from 
a LIME file is provided by the utility 

7„ lime_extract_type limeFile recordType 

which writes the (first) record of the specified 
type to stdout. 

6. USING LDG-SOFTWARE 

In the following sections, an overview of the 
LDG-software architecture is given, followed by 
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a short description of the user interface (called 
Itools) and a sample session. 

6.1. LDG-Software architecture 

Within the LDG community, there is a soft¬ 
ware bundle installable on any computer run¬ 
ning Linux. The bundle contains all the neces¬ 
sary software to work with LDG (assuming you 
own a valid certificate). This includes access to 
the meta data catalogue (MDC) and all partici¬ 
pating storage-elements (SEs). Actual access to 
the SEs is realised by the use of the LGG-2 soft¬ 
ware [ 7 | and access to the MDC by the use of a 
special client-software developed by DESY. This 
complexity is hidden from the user by offering a 
simple set of commands that combines both (see 
the following section). All software can be in¬ 
stalled in a simple way and without the need of 
root privileges. 

Documentation and the software itself can be 
found in [H]. Initially, a packaging system called 
Irpm (which is a kind of rpm tailored for the 
needs of LDG) has to be installed and the loca¬ 
tion where you want to install the software has to 
be defined. After this, installation, initialisation, 
updating or deleting of the software is very easy, 
as only one or a few commands have to be typed 
for each purpose. 

6.2. User tools 

The authors have written a set of easy to use 
command line tools called Itools which are dis¬ 
tributed as part of the LDG-software package 1^. 
On the one hand, the motivation was to simplify 
the usage of the corresponding LCG commands 
within the LDG context by using natural defaults, 
combining sequences of commands, prevent the 
typical user from erroneous use, providing better 
explanation- and error-messages and making the 
software more configurable to personal needs. On 
the other hand, the commands where designed to 
also access the MDG without the need of an ad¬ 
ditional software for the user. In particular, the 
upload of the data to the SE and the upload of 
the metadata to the MDC is combined in a sin¬ 
gle transaction to circumvent inconsistencies in 
case of an error. An overview of the commands 
is given in Table 0 


6.3. Sample session 

A typical session with Itools looks as follows. 
The user starts by taking a look at which ensem¬ 
bles are stored in the MDC by typing: 

"/. 11s —all 

On the first call of an Itool command 
grid-proxy-init will be initiated automatically, 
so that the user is prompted for his passphrase 
(unless the user has already a grid-proxy run¬ 
ning). The output of 11s —all is a list of 
MarkovChainURIs. For each MarkovChainURI 
one can get a list of LFNs of all configurations 
for that URI by typing: 

°/o 11s MarkovChainURI 

One can download the metadata of a configura¬ 
tion by typing: 

y. Iget -m LFN 

After inspection of the metadata, one might want 
to download the actual configuration binary: 

y. Iget LFN 

A user’s guide with the full functional description 
of the commands can also be found at HU. Ta¬ 
ble 0 shows the output of an Iget execution for 
downloading a configuration binary. 

7. SUMMARY 

In this Transnational Access Activity a hard- 
and software infrastructure was set up that is tai¬ 
lored to storing configurations from simulations of 
QCD and that is well integrated into the Interna¬ 
tional Lattice DataGrid activities of the Compu¬ 
tational Hadron Physics community. 
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Table 4 

Sample session, grid-proxy-init is automatically run at the first call of an Uool command. 

y, Iget /grid/ildg/qcdsf/nf2_clover/b5p29kpl3500-16x32/qcdsf.515.00320.lime 
Welcome to the Ltool-command Iget - 
Testing grid-proxy-init: 

Trying to start grid-proxy-init... 

Your identity: /0=GermanGrid/0U=ZlB/CN=Hinnerk Stueben 
Enter GRID pass phrase for this identity: 

Creating proxy . Done 

Your proxy is valid until: Wed Nov 16 03:56:22 2005 

Trying to get binary ... 

Virtual Organisation is ildg 

Executing Icg-cp Ifn:/grid/ildg/qcdsf/nf2_clover/b5p29kpl3500-16x32/qcdsf.515.00320.1 
ime — VO ildg file:/home/stueben/qcdsf.515.00320.lime 

Checking nonzero size of downloaded File ...ok. 
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